Benchmarks Aug 21, 2026 8 min read
We ran 40,000 summaries through twelve models. Here is what it cost.
The cheapest model that clears a quality floor is rarely the one you would guess.
Teo Marchetti
Research
Summarization is the workload most teams start with and the one most teams overpay for. We took forty thousand real call transcripts — anonymised, with permission — and ran every one through the twelve models in the Prism pool.
How we judged quality
A frontier model graded each summary against the transcript on coverage, faithfulness and length, one to a hundred. We spot-checked four hundred grades by hand; the judge and the humans disagreed by more than ten points on 3% of them.
The table
| Model | Quality | Cost per 1,000 summaries |
|---|---|---|
| Claude Sonnet 5 | 91 | $27.40 |
| GPT-5.6 | 90 | $38.10 |
| DeepSeek V4 Pro | 86 | $9.60 |
| Gemini 3.1 Flash | 84 | $4.20 |
| Llama 4 Maverick | 79 | $5.90 |
What it means
If your quality floor is eighty, Gemini 3.1 Flash does the job for a sixth of what a frontier model costs. If your floor is ninety, only two models clear it and the cheaper one is 28% cheaper. The router's job is to know which floor you set and stop paying above it.
Set policy: cheapest-above-80 and this whole table becomes a line in your log.